[auto-techsupport] support techsupport generation on potential memory… #939

stepanblyschak · 2022-02-04T12:07:29Z

… leaks

PR title	state	context
[auto-ts] add memory check
[auto-ts] add memory check

… leaks

Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>

qiluo-msft · 2022-03-10T07:34:43Z

doc/auto_techsupport_and_coredump_mgmt.md

+
+the SONiC techsupport is automatically generated.
+
+```mem_free_threshold``` is there to invoke dump when there is quite small amount of memory left that is needed to successfully execute "show techsupport". This is going to be 200 Mb by default, as at least 80-90 Mb takes "show techsupport" execution.


mem_free_threshold

If suddenly mem_free_threshold < 80 MB, show techsupport will always fail. This invocation will make situation worse. #Closed

@qiluo-msft If user configures too low mem_free_thrshold the "show techsupport" will fail. Currently there is no upper limit of "show techsupport" peak memory consumption, which will make it hard to verify the user does not configure this threshold too low.

This invocation will make situation worse.

It is assumed that the situation is already critical and we try to take the dump as best effort.

BTW, the swap partition would make it at least possible to execute some commands, however, it is not configured in SONiC.

Sorry for misleading. I am asking if the free memory suddenly drop to below 80 MB, show techsupport will always fail. This invocation will make situation worse.

@qiluo-msft Do you mean that if suddenly an application perfroms a very big allocation so that we won't be able to perform "show techsupport". A similar situation might happen when the process leaks memory very quickly so that we don't have enough memory to inoke "show techsupport". In this design the dump collection is best effort, so we don't guaranty that the dump will be actually generated because of that. It is also assumed that if the system is running with that low free memory the situation is already critical (very likelly that reboot will happen) so that additional "show tech" does not help or make it worth, it is making the reboot happen earlier. If the user does want the system to stay with a very low free memory than the user has to disable this feature. Otherwise I don't see a way to handle such a situation.

qiluo-msft · 2022-03-10T07:42:39Z

doc/auto_techsupport_and_coredump_mgmt.md

+    if status != 0 for 10 times within 20 cycles then exec /usr/local/bin/mem_threshold_check_handler"
+```
+
+The action is going to be ran only once the mem_check script detects memory usage above threshold.


once

Do you mean once after each bootup? Considering some fluctuations and finally a memory exhausting, you will keep only the first fluctuation "show techsupport" file? #Closed

@qiluo-msft I mean that the techsupport will run once if the script returns non zero status code for 10 times within 20 cycles. If memory usage is constantly too high for e.g 100 cycles, only 1 dump will be taken. If lets say the memory usage fluctuates, in example, 10 cycles too high in a row and then goes down for 10 cycles and then again up - there is a protection against too frequent auto tech dumps already present in the auto tech infrastructure.

qiluo-msft · 2022-03-10T07:45:09Z

doc/auto_techsupport_and_coredump_mgmt.md

+
+the SONiC techsupport is automatically generated.
+
+```mem_free_threshold``` is there to invoke dump when there is quite small amount of memory left that is needed to successfully execute "show techsupport". This is going to be 200 Mb by default, as at least 80-90 Mb takes "show techsupport" execution.


Mb

Mb -> MB #Closed

@qiluo-msft updated in next patch

qiluo-msft · 2022-03-10T07:45:20Z

doc/auto_techsupport_and_coredump_mgmt.md


 ### Config DB

 #### AUTO_TECHSUPPORT Table
 ```
 key                    = "AUTO_TECHSUPPORT|global"  
 state                  = "enabled" / "disabled"    ; Enable this to make the Techsupport Invocation event driven based on core-dump generation
+available_mem_threshold = 1*2DIGIT                 ; Memory threshold; 0 to disable techsupport invocation on mem leak.
+min-available-mem      = 1*5DIGIT                  ; Minimum free memory amount in Kb when techsupport will be executed.


Kb

Kb -> KB #Closed

@qiluo-msft updated in next patch

qiluo-msft · 2022-03-10T07:45:58Z

doc/auto_techsupport_and_coredump_mgmt.md


 ### Config DB

 #### AUTO_TECHSUPPORT Table
 ```
 key                    = "AUTO_TECHSUPPORT|global"  
 state                  = "enabled" / "disabled"    ; Enable this to make the Techsupport Invocation event driven based on core-dump generation
+available_mem_threshold = 1*2DIGIT                 ; Memory threshold; 0 to disable techsupport invocation on mem leak.
+min-available-mem      = 1*5DIGIT                  ; Minimum free memory amount in Kb when techsupport will be executed.


Use underscore instead of dash? #Closed

@qiluo-msft Right, updated in next patch

qiluo-msft · 2022-03-10T07:46:32Z

doc/auto_techsupport_and_coredump_mgmt.md


 ### Config DB

 #### AUTO_TECHSUPPORT Table
 ```
 key                    = "AUTO_TECHSUPPORT|global"  
 state                  = "enabled" / "disabled"    ; Enable this to make the Techsupport Invocation event driven based on core-dump generation
+available_mem_threshold = 1*2DIGIT                 ; Memory threshold; 0 to disable techsupport invocation on mem leak.
+min-available-mem      = 1*5DIGIT                  ; Minimum free memory amount in Kb when techsupport will be executed.


Kb

The same #Closed

If always > 1MB, use MB as unit?

@qiluo-msft Agree, updated in next patch

qiluo-msft · 2022-03-10T07:50:31Z

doc/auto_techsupport_and_coredump_mgmt.md

@@ -226,27 +298,43 @@ module sonic-auto_techsupport {
 #### AUTO_TECHSUPPORT_DUMP_INFO Table


Table

Is there an entry number limit in this table? #Closed

@qiluo-msft There is a rate limiting mechanism already available in auto techsupport infrastructure that limits the number of dumps that can be generated.

qiluo-msft · 2022-03-10T07:55:51Z

doc/auto_techsupport_and_coredump_mgmt.md

@@ -36,13 +37,20 @@ Currently, techsupport is run by invoking `show techsupport` either by orchestra

 However if the techsupport invocation can be made event-driven based on core dump generation, that would definitely improve the debuggability. That is the overall idea behind this HLD. All the high-level requirements are summarized in the next section

+Another use case is to gather more information about the system in case there is a memory usage threshold crossed.
+SONiC dump generated after system reboots due to out of memory is not enough for debugging the issue
+as all the information about processes and their mem usage, smaps (/proc/PID/smaps) is lost.


lost

Instead of invoking a full show techsupport, can we just save volatile information before reboot? For example, it makes no sense to save syslog during the memory pressure time. We could collect them after reboot. #Closed

@qiluo-msft The user can configure the "since" parameter in CFG_DB and achive that result. If "since" is set to "0 sec" that would mean that no logs, sai, swss recs are recorded.

qiluo-msft · 2022-03-10T07:58:36Z

doc/auto_techsupport_and_coredump_mgmt.md

+    if status != 0 for 10 times within 20 cycles then exec /usr/local/bin/mem_threshold_check_handler"
+```
+
+The action is going to be ran only once the mem_check script detects memory usage above threshold.


threshold

If auto invoking too many times, do we have file rotation? Do we only rotate the auto dump files and keep user manual dump? #Closed

@qiluo-msft There are "rate-limit-interval", "max-techsupport-limit" parameters in AUTO_TECHSUPPORT table which are already part of auto tech infrastructure that limit the number of dumps. This is only applicable for automatically generated dumps.

I see. Is auto rotation covered by design?

Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>

liat-grozovik · 2022-03-20T08:29:13Z

@stepanblyschak please update PR description with the PRs list covering this feature.

- What I did Implemented memory threashold check in auto techsupport feature according to sonic-net/SONiC#939. - How I did it Added two scripts. The check script and the handler script. Few modifications made in auto tech implementation. UT added. - How to verify it Run the action script and the handler script on the switch. Run UT. Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>

#### Why I did it To support automatic techsupport invokation in case memory usage is too high. #### How I did it Implemented according to sonic-net/SONiC#939 #### How to verify it UT, manual test on the switch. *DEPENDS* on sonic-net/sonic-utilities#2116

Backport of #2116 - What I did Implemented memory threashold check in auto techsupport feature according to sonic-net/SONiC#939. - How I did it Added two scripts. The check script and the handler script. Few modifications made in auto tech implementation. UT added. - How to verify it Run the action script and the handler script on the switch. Run UT. Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>

#### Why I did it To support automatic techsupport invokation in case memory usage is too high. #### How I did it Implemented according to sonic-net/SONiC#939 #### How to verify it UT, manual test on the switch. *DEPENDS* on sonic-net/sonic-utilities#2116

- What I did Implemented memory threashold check in auto techsupport feature according to sonic-net/SONiC#939. - How I did it Added two scripts. The check script and the handler script. Few modifications made in auto tech implementation. UT added. - How to verify it Run the action script and the handler script on the switch. Run UT. Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>

stepanblyschak and others added 8 commits January 27, 2022 02:44

[auto-techsupport] support techsupport generation on potential memory…

ea5bc56

… leaks

Update auto_techsupport_and_coredump_mgmt.md

9cdfc05

Update auto_techsupport_and_coredump_mgmt.md

991eba4

Update auto_techsupport_and_coredump_mgmt.md

e178596

Update auto_techsupport_and_coredump_mgmt.md

86c46a3

[auto techsupport] review comments

ad03b32

Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>

Update auto_techsupport_and_coredump_mgmt.md

c7b9093

Update auto_techsupport_and_coredump_mgmt.md

669409c

qiluo-msft self-requested a review March 7, 2022 17:28

qiluo-msft reviewed Mar 10, 2022

View reviewed changes

review comments fixes

6a588d7

Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>

qiluo-msft approved these changes Mar 19, 2022

View reviewed changes

liat-grozovik approved these changes Mar 20, 2022

View reviewed changes

liat-grozovik merged commit 61e6c87 into sonic-net:master Mar 20, 2022

This was referenced Mar 26, 2022

[auto-ts] add memory check sonic-net/sonic-utilities#2116

Merged

[auto-ts] add memory check sonic-net/sonic-buildimage#10433

Merged

This was referenced Sep 29, 2022

[202205][auto-ts] add memory check stepanblyschak/sonic-utilities#58

Closed

[202205][auto-ts] add memory check (#2116) sonic-net/sonic-utilities#2413

Merged

stepanblyschak mentioned this pull request Oct 6, 2022

[202205][auto-ts] add memory check (#10433) sonic-net/sonic-buildimage#12291

Merged

7 tasks

liat-grozovik mentioned this pull request Oct 11, 2022

202205 Release Notes #1087

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[auto-techsupport] support techsupport generation on potential memory… #939

[auto-techsupport] support techsupport generation on potential memory… #939

stepanblyschak commented Feb 4, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited

stepanblyschak Mar 16, 2022

qiluo-msft Mar 17, 2022

stepanblyschak Mar 18, 2022

qiluo-msft Mar 10, 2022 •

edited

stepanblyschak Mar 16, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited

stepanblyschak Mar 16, 2022

qiluo-msft Mar 10, 2022 •

edited

stepanblyschak Mar 16, 2022

qiluo-msft Mar 10, 2022 •

edited

stepanblyschak Mar 16, 2022

qiluo-msft Mar 10, 2022 •

edited

qiluo-msft Mar 10, 2022

stepanblyschak Mar 16, 2022

qiluo-msft Mar 10, 2022 •

edited

stepanblyschak Mar 16, 2022

qiluo-msft Mar 10, 2022 •

edited

stepanblyschak Mar 16, 2022

qiluo-msft Mar 10, 2022 •

edited

stepanblyschak Mar 16, 2022

qiluo-msft Mar 17, 2022

stepanblyschak Mar 18, 2022

liat-grozovik commented Mar 20, 2022


		the SONiC techsupport is automatically generated.

		```mem_free_threshold``` is there to invoke dump when there is quite small amount of memory left that is needed to successfully execute "show techsupport". This is going to be 200 Mb by default, as at least 80-90 Mb takes "show techsupport" execution.

		@@ -226,27 +298,43 @@ module sonic-auto_techsupport {
		#### AUTO_TECHSUPPORT_DUMP_INFO Table

[auto-techsupport] support techsupport generation on potential memory… #939

[auto-techsupport] support techsupport generation on potential memory… #939

Conversation

stepanblyschak commented Feb 4, 2022 • edited

qiluo-msft Mar 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Mar 10, 2022 • edited

Choose a reason for hiding this comment

stepanblyschak Mar 16, 2022 • edited

Choose a reason for hiding this comment

qiluo-msft Mar 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Mar 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Mar 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Mar 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Mar 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Mar 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft Mar 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liat-grozovik commented Mar 20, 2022

stepanblyschak commented Feb 4, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited

stepanblyschak Mar 16, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited

qiluo-msft Mar 10, 2022 •

edited